DOC-404: Document orphaned containers after Docker Compose stop - #894
DOC-404: Document orphaned containers after Docker Compose stop#894quetzalliwrites wants to merge 2 commits into
Conversation
Add an FAQ entry explaining that Docker Compose's default 10s SIGTERM-to-SIGKILL grace period can be too short for LocalStack to finish tearing down auxiliary containers (including k3d containers backing an EKS cluster), leaving them behind as orphans, and how to raise stop_grace_period to avoid it. The CLI is not affected since it waits for shutdown to finish. Cross-link the same behavior from the EKS docs where k3d-backed cluster creation is introduced.
Deploying localstack-docs with
|
| Latest commit: |
ca6574b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7d4e2d3c.localstack-docs.pages.dev |
| Branch Preview URL: | https://doc-404-doc-faq-addition-of.localstack-docs.pages.dev |
|
@nik-localstack @pinzon heads up, looks like this PR is still pending eng review |
nik-localstack
left a comment
There was a problem hiding this comment.
Thanks for putting this together, the entry describes a real problem, and it came up in an internal discussion recently too.
Some things from that are worth folding in:
- There's a second setting involved, and it isn't in our docs. Alongside docker-compose stop_grace_period, LocalStack applies its own
SHUTDOWN_TIMEOUT(5 seconds by default) to its cleanup, and that one runs out first. Raising onlystop_grace_perioddoesn't give the cleanup more time and it has a side effect worth warning about: docker compose stop then takes the full three minutes before the container exits, whereas raising both settings gets a clean shutdown in well under a minute. The inline suggestions reflect that.SHUTDOWN_TIMEOUTisn't listed in the configuration reference today, so this entry would be the first place we mention it. We should probably expose it in the configuration page. - It isn't only EKS. The same shutdown budget is shared by every service that manages containers. ECS is the closest example, where a task's own stop timeout can consume the entire budget by itself.
I think we need more eyes before merging: shutdown behavior is being actively discussed internally right now, and how these settings interact may change. I don't think that blocks the changes but we should consider what should be advertised as a permanent solution and what not.
cc @purcell who was involved in this discussion about shutdown
btw I have added some inline suggestions but it may not be what we want to document in the end, so feel free to disregard them.
|
Hey @purcell we need confirmation from ENG about whether this docs ticket request is still needed or if it's been cancelled, per @nik-localstack earlier Q: If you're not the right person to confirm, can you let us know who is? thx ! |
- Generalize the FAQ entry away from EKS/k3d specifics (Lambda, ECS, RDS, and EKS all start auxiliary containers); keep the k3d-specific framing in the EKS doc's cross-referencing note instead. - Explain both timing windows that can cause orphaned containers: Docker Compose's stop_grace_period and LocalStack's own SHUTDOWN_TIMEOUT, and note both need to be raised together. - Clarify the CLI callout refers to lstk specifically, not the legacy localstack CLI. - Fix the cleanup instruction to actually say how to find the containers (docker ps) before removing them.
|
Thanks @nik-localstack, while we wait confirmation on whether this ticket is cancelled, I still went ahead and pushed
One thing worth double-checking before merge (independent of the cancellation question): I couldn't find |
purcell
left a comment
There was a problem hiding this comment.
This is all accurate and reasonable, so approving this in principle, with the few proposed edits below to fix some formatting and idioms.
|
|
||
| Raise them together. If you raise only `stop_grace_period`, LocalStack still gives up on its cleanup after 5 seconds and the container then stays up until the grace period runs out, so every docker compose stop waits the full three minutes. | ||
|
|
||
| If a previous run already left containers behind, run docker ps and remove the ones LocalStack started before starting it again. |
There was a problem hiding this comment.
| If a previous run already left containers behind, run docker ps and remove the ones LocalStack started before starting it again. | |
| If a previous run already left containers behind, run `docker ps` and remove the ones LocalStack started before starting it again. |
|
|
||
| By default, LocalStack gets only a few seconds. Docker Compose sends `SIGTERM` to the container and follows up with `SIGKILL` once `stop_grace_period` expires (10 seconds by default), and LocalStack stops waiting for its own cleanup after `SHUTDOWN_TIMEOUT` (5 seconds by default). Whichever window runs out first, LocalStack is stopped mid-cleanup and the containers survive as orphans. Cleanup that takes longer than a few seconds is the most likely to be cut short. | ||
|
|
||
| To give LocalStack enough time to shut down cleanly, raise `stop_grace_period` on the LocalStack service in your `docker-compose.yml` (for example, `3m`): |
There was a problem hiding this comment.
| To give LocalStack enough time to shut down cleanly, raise `stop_grace_period` on the LocalStack service in your `docker-compose.yml` (for example, `3m`): | |
| To give LocalStack enough time to shut down cleanly, increase `stop_grace_period` on the LocalStack service in your `docker-compose.yml` (for example, `3m`): |
| # ... | ||
| ``` | ||
|
|
||
| Raise them together. If you raise only `stop_grace_period`, LocalStack still gives up on its cleanup after 5 seconds and the container then stays up until the grace period runs out, so every docker compose stop waits the full three minutes. |
There was a problem hiding this comment.
| Raise them together. If you raise only `stop_grace_period`, LocalStack still gives up on its cleanup after 5 seconds and the container then stays up until the grace period runs out, so every docker compose stop waits the full three minutes. | |
| Increase both settings. If you increase only `stop_grace_period`, LocalStack still gives up on its cleanup after 5 seconds and the container then stays up until the grace period runs out, so every `docker compose stop` waits the full three minutes. |
|
|
||
| When LocalStack shuts down, it cleans up the auxiliary containers it started for services such as Lambda, ECS, RDS or EKS. If it is stopped before that cleanup finishes, those containers are left running. | ||
|
|
||
| By default, LocalStack gets only a few seconds. Docker Compose sends `SIGTERM` to the container and follows up with `SIGKILL` once `stop_grace_period` expires (10 seconds by default), and LocalStack stops waiting for its own cleanup after `SHUTDOWN_TIMEOUT` (5 seconds by default). Whichever window runs out first, LocalStack is stopped mid-cleanup and the containers survive as orphans. Cleanup that takes longer than a few seconds is the most likely to be cut short. |
There was a problem hiding this comment.
| By default, LocalStack gets only a few seconds. Docker Compose sends `SIGTERM` to the container and follows up with `SIGKILL` once `stop_grace_period` expires (10 seconds by default), and LocalStack stops waiting for its own cleanup after `SHUTDOWN_TIMEOUT` (5 seconds by default). Whichever window runs out first, LocalStack is stopped mid-cleanup and the containers survive as orphans. Cleanup that takes longer than a few seconds is the most likely to be cut short. | |
| By default, LocalStack gets only a few seconds in which to shut down. Docker Compose sends `SIGTERM` to the container and follows up with `SIGKILL` once `stop_grace_period` expires (10 seconds by default), and LocalStack stops waiting for its own cleanup after `SHUTDOWN_TIMEOUT` (5 seconds by default). Whichever window runs out first, LocalStack is stopped mid-cleanup and the containers survive as orphans. Cleanup that takes longer than a few seconds is the most likely to be cut short. |
| services: | ||
| localstack: | ||
| image: localstack/localstack-pro:latest | ||
| stop_grace_period: 3m |
There was a problem hiding this comment.
| stop_grace_period: 3m | |
| stop_grace_period: 180s |
Also, writing this period as seconds makes it more obviously match the number below.
Summary
stop_grace_periodto avoid it.Linear ticket
https://linear.app/localstack/issue/DOC-404/doc-faq-addition-of-why-are-some-containers-left-behind-after-i-stop
Test plan
npx astro buildcompletes successfully (401 pages built)eks.mdxandfaq.mdx, are valid